On the Distribution of Mutual Information

In the early years of information theory, mutual information was defined as a random variable, and error probability bounds for communication systems were obtained in terms of its probability distribution. We advocate a return to this perspective for a renewed look at information theory for general channel models and finite coding blocklengths. For capacityachieving inputs, we characterize the mutual information random variables for several important channel models, including the discrete memoryless binary symmetric channel (BSC), the discrete-time complex additive white Gaussian noise (AWGN) channel, and the discrete-time Rayleigh fading channel with static, flat fading known perfectly to the decoder. Combined with a result known as Feinstein’s lemma and its extensions, these characterizations can be employed to obtain bounds on the maximal block error probability of channel codes operating over these channels. Such bounds appear to be particularly useful when operating at transmission rates near the average mutual information.